Results and Observations from the Evaluation of a Telephony- based Conversational Information Retrieval System

This paper presents the observations and results from the evaluation of the Program Guide Information System (PGIS), which was built as the CATCH-2004 Helsinki demonstrator under the European Union’s 5 Framework Research Programme [1]. The technical implementation is the outcome of the integration of several technologies, including Automatic Speech Recognition and Natural Language Understanding. The paper gives an overview of the architecture and the technical implementation of the system. To achieve a high quality conversational interface implementation, the developed system was tested in several rounds. Both objective and subjective tests were run in an iterative manner: after each test cycle the collected data was analysed and used to improve the overall system and the individual modules’ performance further. Users’ opinions about the usability of the system were taken into account in tuning the user interface, i.e. modifying system prompting, dialogue flow, feedback mechanism, etc. The methodology of the evaluation perspectives is introduced and the main results and observations of the test runs are presented in more details.